1
Transforming EHR Interactions Using Voice Assistants
Session 164, February 13, 2019
Yaa Kumah-Crystal, MD MPH
Timothy Coffman
Vanderbilt University Medical Center
2
Yaa Kumah-Crystal, MD MPH
Discloses the following relevant relationship with commercial
interests:
Research partnership with Nuance Communications
Use of NLP libraries for research and development
Conflict of Interest
3
Timothy Coffman
Discloses the following relevant relationship with commercial
interests:
Research partnership with Nuance Communications
Use of NLP libraries for research and development
Conflict of Interest
4
Principles employed in creating a voice user interface for the
electronic health record
Challenges faced in data capture
Challenges faced in data presentation
Assessment
Implementation and future plans
Agenda
5
Discuss the value of artificial intelligent (AI)-enabled natural
language processing and understanding technologies in the
advancement of healthcare
Discuss the limitations of current keyboard and mouse workflows
in the EHR
Describe the usability design considerations, information request
mapping and artificial intelligence/natural language understanding
platform built to process voice requests
Discuss the provider cohort experience and outcomes around
satisfaction
Learning Objectives
6
Informal Poll
Own a smart phone
Use your smart phone voice assistant
Siri, Google assistant, Bixby
Own a voice assistant device
Alexa, Google home, Apple Homepod
7
Why Voice?
8
Documentation Modalities
Writing
~13
WPM
Typing
~40
WPM
Speech
~150
WPM
9
Why not Voice?
10
Macros
Physical exam
Insert normal physical exam
11
Vanderbilt Innovations Portfolio Research
12
Given
Installed EHR
EHR vendor has existing partnership with natural language
services provider (including dictation)
Existing in-house innovation capacity
Infrastructure supporting custom development
Team in place with necessary SW development skills
Mission to develop expertise in new ways to interact with the EHR
Decision
Develop new software to integrate EHR with additional natural
language services
Solutions
13
Well really, how how hard can it be?
13
14
Oh...
15
Demonstration
16
17
VEVA Processing
NLP
User Query
Speech to text
Intent Fulfillment
Text to speech
Results Display
What is her
A1C?
Sally’s A1C
is 8.6
18
VEVA Architecture
19
EHR’s FHIR implementation does not yet support all desired data
Additional development required for each new query
New phrasings of questions requiring updates to language model
Microphone and speaker issues in a virtual desktop environment
Latency
Technical Limitations
20
Focus on developing demonstration of capabilities supporting
One (1) specialty clinic
Handful of general responses
Aligned with FHIR (DSTU2) resources
Some EHR-proprietary services
Build in capability to extend with
Additional capabilities for other specialties
General responses, discovered through user feedback
FHIR (STU3) and FHIR (R4) resources as available
FHIR-based alternatives to proprietary services
Planning for Extension
21
MVP
Rapid Iteration
Vanderbilt as a Design Shop
Technical Considerations
22
Integrating voice response involves real-time services
Audio streaming capabilities
Virtual desktops introduce challenges
Automatic connections
Latency
Varying hardware installed at point-of-use
Individual laptop microphones
High-end dictation microphones
All of the above
Infrastructure Challenges
23
“While voice is a natural interface for humans, this may be the first
time we are using it as an interface to have a computer
communicate back to us like this. Different rules may apply.
Design Considerations
24
Brevity vs. Clarity
“Blood pressure is 120 millimeters of mercury over 80
millimeters of mercury ”
“Blood pressure is 120 over 80”
Uncanny Valley of Words
I ate three big red apples
I ate red big three apples
Quantity, Opinion, Size, Age, Shape, Color, Material,
Purpose
Design Considerations
25
Use Cases
26
Preparing to see a patient in a provider workroom
Large screen
PHI private
Use Case A
27
Reviewing information with patient present
Large screen
PHI with patient 3
rd
party
Use Case B
28
In transition in a private environment (commute)
Small Screen (mobile)- limited visibility
Primarily verbal navigation
PHI private
Use Case C
29
In transition in a public environment (hallway)
Small screen (mobile)
PHI considerations
(aloud only with headphones?)
Use Case D
30
Usability Study
31
Setting: Vanderbilt University Medical Center
Cohort: 14 Pediatric Endocrinology Providers
Female (10) | Male (4)
Mean age 39, range 29-65
Study Design
32
Usability Assessment
6 Skills Evaluated
Training: Hello / What can you do
Summary
A1C
Weight
Blood Pressure
Health Maintenance
Study Design
33
Value of information communicated
Length of content communicated
Value of graphical display
Overall thoughts about voice interactions
Feedback
34
System Usability Scale: Positively Worded
Strongly Disagree
Mean 4.12 (SD .46)
3.93 (SD 1.20)
4.50 (SD .52)
3.79 (SD .70)
4.00 (SD .79)
4.43 (SD .51)
Strongly Agree
35
System Usability Scale: Negatively Worded
Mean 1.67 (SD .66)
Strongly Disagree
Strongly Agree
1.36 (SD .84)
1.71 (SD .83)
1.79 (SD 1.19)
1.64 (SD .74)
1.86 (SD 1.03)
36
Mean System Usability Scale score 80.7 (highly usable)
Scores of >68 are considered above average
Summary
Relevant medications, other problems
Weight
Growth chart
Blood pressure
Percentile information
User Feedback
37
Willingness to Use System
36%
64%
NoYes
4 count9 count
2
1
1
38
Incorporation of User Feedback
Desktop Time Comparison
Scalability
Alerts and Decision Support
Next Steps
39
Understand and measure provider frustration with EHR
Create the business case for optimizing their experience and
saving providers time within their EHR workflow
Assemble a cross-functional team to build the overarching model
taking into consideration the providers’ pain points, workflow, and
information needs
Build and iterate a voice assistant prototype while users test and
provide feedback
Understand information theory and map queries and content to
satisfy users’ needs
Best Practices/Recommendations
40
An Interesting Study About Voice
Nationwide Children's Hospital in Columbus, Ohio
41
It looks like you have a question.
Yes
Go away Clippy!
Questions
Please remember to complete the online session evaluation
42
Contact Us